Why is late-interaction reranking typically applied to a small candidate set rather than the entire collection?

Late interaction is expensive in a way that scales multiplicatively with document length. For a single dense vector comparison, the cost is one dot product of length d. For MaxSim, the cost is q * t dot products of length d, where q is the number of query tokens and t is the number of document tokens. If q is 20 and t is 200, that is 4000 dot products per document - roughly 4000 times the work of a single dense comparison. If you tried to apply that to an entire collection of 100 million documents, you would be doing 400 billion dot products per query, which is not feasible at any reasonable latency. Even if you could index the token vectors to avoid the full scan, the fundamental issue is that MaxSim is not decomposable into a single vector operation - you cannot precompute a single embedding that captures the result of MaxSim against an arbitrary query. So you cannot use HNSW or any other single-vector ANN index to accelerate it. The only practical way to use late interaction is to first reduce the candidate set with a cheap retriever, then apply MaxSim to the small set.

The mechanism of the pipeline is therefore always the same shape: a first-stage retriever (dense, sparse, or hybrid) produces a candidate set of a few hundred documents using an index that supports sub-linear search, and then a late-interaction reranker scores those candidates exactly and returns the top-k. The first stage optimizes for recall - it must not miss relevant documents - while the second stage optimizes for precision - it must order the candidates correctly. This is a classic retrieve-then-rerank architecture, and it works because the two stages have complementary cost profiles. The first stage is cheap per document but must scan a large corpus; the second stage is expensive per document but only sees a small set. The candidate set size is the key tuning parameter: too small and the first stage's recall becomes the bottleneck (the reranker cannot recover documents that were never retrieved); too large and the reranker's cost dominates the latency budget. Typical values are 50-200 candidates, but the right number depends on the first stage's recall and the latency budget.

The trade-off is end-to-end recall against latency, and the first-stage recall ceiling is the subtle part. A common mistake is to tune the reranker aggressively while ignoring the first stage, then wonder why end-to-end recall plateaus at 0.85. The reranker cannot rank documents it never sees. The second common mistake is to set the candidate set size too small to save latency, which quietly caps recall. Measuring first-stage recall in isolation - by comparing the candidate set against exact ground truth - is the only way to know whether the bottleneck is the retriever or the reranker. The alternative to a fixed candidate set is an adaptive one: retrieve more candidates for hard queries (e.g. long queries, rare terms) and fewer for easy ones. This is harder to implement but can meaningfully reduce average latency without sacrificing recall. Version note: Qdrant's prefetch and fusion API lets you express a two-stage pipeline in a single query, but the exact nesting semantics and the supported comparators have changed across releases - verify against your version.

Version-dependent: the prefetch/fusion API used here is qdrant-client 1.10+. On older versions, two-stage retrieval was typically implemented by making two separate requests in the application layer - one to get candidates and one to rerank them - which has the same effect but a different code shape and different round-trip latency. The performance characteristics of the two approaches can differ because the in-engine version avoids serializing the candidate set over the network.